This is an automated cherry-pick of #4570

Signed-off-by: xufei [email protected]

What problem does this PR solve?

Issue Number: close #4451

Problem Summary:

What is changed and how it works?

Actually, there are 2 possible solutions

• Consider local delta memory usage when estimate the result_size_bytes in aggregator on each execute of executeOnBlock
• In Aggregator::mergeAndConvertToBlocks, check the result_size_bytes, and if it exceeds the threshold, convert all the hash table into two-level hash table

For the first solution, converting hash table to two-level hash table can be done by each threads in the first stage of ParallelAggregating, and for the second solution, this converting things are executed in 1 threads.

I've done some test for both solutions, and found first solution has ~20% perfomance gain compared to the second solution for TPCH q15. So I choose the first solution.

Why not enable two-level hash table by default for all the cases:

• not all the agg method support two-level hash table
• test results shows that always use two-level hash table is slower if the table data is small, and the size of agg results is small.

Testing query:
select count(*),id from test group by id;
test has 65536 rows, all rows have the same id, that is to say the above query return only 1 row
Testing cluster has only 1 TiFlash, and the cpu has 36 core, each query is running 1000 times.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

None